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Abstract 

We are interested in the study of models describing the evolution of a polymor- 
phic population with mutation and selection in the specific scales of the biological 
framework of adaptive dynamics. The population size is assumed to be large and the 
mutation rate small. We prove that under a good combination of these two scales, the 
population process is approximated in the long time scale of mutations by a Markov 
pure jump process describing the successive trait equilibria of the population. This 
process, which generalizes the so-called trait substitution sequence, is called polymor- 
phic evolution sequence. Then we introduce a scaling of the size of mutations and we 
study the polymorphic evolution sequence in the limit of small mutations. From this 
study in the neighborhood of evolutionary singularities, we obtain a full mathematical 
justification of a heuristic criterion for the phenomenon of evolutionary branching. 
To this end we finely analyze the asymptotic behavior of 3-dimensional competitive 
Lotka-Volterra systems. 
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1 Introduction 

We consider an asexual population in which each individual's ability to survive and repro- 
duce is characterized by a quantitative trait, such as the size, the age at maturity, or the 
rate of food intake. Evolution, acting on the trait distribution of the population, is the 
consequence of three basic mechanisms: heredity, which transmits traits to new offsprings, 
mutation, driving a variation in the trait values in the population, and selection between 
these different trait values, which is due to the competition between individuals for limited 
resources or area. Adaptive dynamics models aim at studying the interplay between these 
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different mechanisms [2H [25], [2?]. Our approach is based on a microscopic individual- 
based model that details the ecological dynamics of each individual. Prom the simulated 
dynamics of this process initially issued from a monomorphic population, we observe that 
it is essentially single-modal centered around a trait that evolves continuously, until some 
time where the population divides into two separate sub-populations that are still in in- 
teraction but are centered around distinct traits at a distance increasing with time. This 
phenomenon, called Evolutionary Branching, is thought to be a possible explanation of 
phenotypic separation without geographic separation [7J. (One speaks about sympatric 
speciation though the population is asexual). Our aim in this paper is to understand the 
dynamics of the process in long time scales and to highlight the evolutionary branching 
phenomenon. In particular, we want to prove the conjecture stated by Metz et al. [26J 
and giving conditions on the parameters of the model allowing one to predict whether 
evolutionary branching will occur or not. 

To this aim, we follow the basic description of adaptive dynamics based on the bi- 
ologically motivated assumptions of rare mutations and large population. Under these 
assumptions, we prove that the microscopic process describing the ecological dynamics can 
be approximated by a Markov pure jump process on the set of point measures on the trait 
space. The transitions of this process are given by the long time behavior of competitive 
Lotka-Volterra systems. They describe the succession of mutant invasions followed by a 
fast competition phase between the mutant population and the resident one. In the mu- 
tation time scale, and for large populations, the successful traits in the competition are 
given by the nontrivial equilibria of Lotka-Volterra systems which model the dynamics of 
the sizes of each sub-population corresponding to each resident or mutant trait. We thus 
generalize the situation introduced in [26] and mathematically developed in [4], where the 
parameters of the model prevent the coexistence of two traits. In that case, the micro- 
scopic model converges to a monomorphic (one trait support) pure jump process, called 
Trait Substitution Sequence (TSS). This limit involves a timescale separation between the 
mutations and the population dynamics driving the competition between traits. 

In this article, we relax the assumption of non-coexistence and obtain a polymorphic 
evolution sequence (PES), allowing coexistence of several traits in the population, from 
the same microscopic model described in Section [2l In Section 12.31 we introduce the 
competitive symmetric Lotka-Volterra systems describing the competition between traits. 
We prove in Section l2~4l that the PES takes the form of a Markov jump process on the set of 
measures on the trait space X that are finite sums of Dirac masses with positive weights, 
and we characterize the transitions of this process in terms of the long time behaviour 
of competitive Lotka-Volterra systems. In Section [31 we explain why the assumptions 
ensuring the convergence to the PES are satisfied as long as no more than two traits 
coexist. In this case, the dynamics of the PES can be explicitely characterized. Next 
(Section 0J, we study the transition from a monomorphic population to a stable dimorphic 
population, and give a full mathematical justification of the criterion for evolutionary 
branching proposed in [26], under the assumption of small mutation effects. To this end, 
we first show in Sections 14. II and l4~2l that . away from evolutionary singularities, the support 
of the PES stays monorphic and converges to an ODE known as the "canonical equation" [8]. 
Finally, in Section 14.31 we characterize the situations where evolutionary branching occurs 
by specializing to our situation the results of Zeeman [30] on the asymptotic behavior of 
3-dimensional competitive Lotka-Volterra systems. 
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Let us insist on the importance of the limits. Here we are concerned by the combination 
of the limits of large populations and rare mutations, followed by a limit of small mutations. 
An alternative approach would be first to study the limit of large population alone, giving in 
the limit an integro-differential partial differential equation for the density of traits [5]; and 
next to study a limit of small mutations on this equation with a proper time scaling that 
would lead to some dynamics on the set of finite sums of Dirac masses on the trait space. 
The second part of this program has already been partly studied in [9] in a specific model, 
but is related to difficult problems on Hamilton- Jacobi equations with constraints [2]. 
In this case, evolutionary branching is numerically observed, but not yet fully justified. 
Another approach would be to combine the three limits we consider directly at the level 
of the microscopic model, allowing one to study the evolutionary process on several time 
scales [3]. This requires a finer analysis of the invasion and competition phases after the 
appearance of a new mutant. Note that all these approaches are based on the same idea 
of separation between the time scales of mutation and competition. 

2 Models and Polymorphic Evolution Sequence (PES) 

Let us introduce here the main models on which our approach is based. 
2.1 The individual-based model 

The microscopic model we use is an individual-based model with density-dependence, which 
has been already studied in ecological or evolutionary contexts by many authors [T2l 15]. 

The trait space X is assumed to be a compact subset of E , I > 1. For any x,y G X, 
we introduce the following biological parameters 

\{x) £ M+ is the rate of birth from an individual holding trait x. 

n(x) G M+ is the rate of "natural" death for an individual holding trait x. 

r{x) := A(x) — fi(x) is the "natural" growth rate of trait x. 

K € N is a parameter scaling the population size and the resources. 

£ JJ + ^ the competition kernel representing the pressure felt by an individual hold- 
ing trait x from an individual holding trait y. It is not assumed to be a symmetric 
function. 

uxp(x) with uk,p(x) € (0,1], is the probability that a mutation occurs in a birth from 
an individual with trait x. Small uk means rare mutations. 

m(x, h)dh is the law of h = y — x, where the mutant trait y is born from an individual 
with trait x. Its support is a subset of X — x = {y — x : y £ X}. 

We consider, at any time t > 0, a finite number N t of individuals, each of them holding 
a trait value in X. Let us denote by xi, . . . , x^ t the trait values of these individuals. 
The state of the population at time t > 0, rescaled by K, is described by the finite point 
measure on X 




(2.1) 



i=i 
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where S x is the Dirac measure at x. Let (v, /} denote the integral of the measurable 
function / with respect to the measure v and Supp(z^) denote its support. 
Then = j& and for any x G X, the positive number (v^,li x \) is called the 

density at time t of trait x. 

Let M.f denote the set of finite nonnegative measures on X, equipped with the weak 
topology, and define 

M K = > ' 5 Xi : n > 0, x lt . . . , x n G X 



i=i 



An individual holding trait x in the population vjr gives birth to another individual with 
rate X(x) and dies with rate 

f 1 Nt 

fi(x) + / a{x,y)vf(dy)=fi(x) + — ^2a(x,Xi). 

^ i=i 

The parameter K scales the strength of competition, thus allowing the coexistence of more 

individuals in the population. A newborn holds the same trait value as its progenitor with 

probability 1 — ukp{x), and with probability ukp(x), the newborn is a mutant whose trait 

value y is chosen according to y = x + h, where h is a random variable with law m(x, h)dh. 

In other words, the process (u^,t > 0) is a A^-valued Markov process with infinitesimal 

generator defined for any bounded measurable functions (j) from M. K to R by 

L K 4>(v) = (j> (v + ^ - cf>(u)] (1 - u K p{x))\{x)Ku(dx) 

+ j( ( (j) { v H — — ( f ) ( l/ )] UKp{x)\(x)m(x,h)dhKu(dx) 
J x Jw V V K J J 

+ L ( v ~ ¥) ~ ^ {u) ) {^ {x) + Sx yHdy) ) Ku{dx) - (2 - 2) 

For v G A4 K , the integrals with respect to Kv(dx) in (12. 2\i correspond to sums over 
all individuals in the population. The first term (linear) describes the births without 
mutation, the second term (linear) describes the births with mutation, and the third term 
(non-linear) describes the deaths by oldness or competition. The density-dependent non- 
linearity of the third term models the competition in the population, and hence drives the 
selection process. 

Let us denote by (A) the following three assumptions 

(Al) A, fj, and a are measurable functions, and there exist \,ft,a < +oo such that 

A(-) < A, fi(-) < and a(-, •) < a. 

(A2) r(x) = X(x) — fj,(x) > for any x € X, and there exists a > such that a < a(-, •). 

(A3) There exists a function m : R — > M + such that m(x, h) < fh(h) for any x G X and 
h G M 1 , and / m(h)dh < oo. 

For fixed K, under (Al) and (A3) and assuming that E((z^,l}) < oo, the existence and 
uniqueness in law of a process on A4 K ) with infinitesimal generator L K has been 

proved in [12]. Assumption (A2) prevents the population to explode and to go extinct too 
fast. 
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2.2 An example 



The birth-death-competition-mutation process described above has been heuristically stud- 
ied in various ecological or evolutionary contexts. Let us illustrate the phenomenon of evo- 
lutionary branching we are interested in with a simple example, adapted from a classical 
model (Roughgarden [29], Dieckmann and Doebeli [7]). In this model, there is a single 
optimal trait value for the birth rate and a symmetric competition kernel. The parameters 
are the following: 

X = [-2,2\; fi(x)=0; p(x) = p, 

X(x) = exp(-x 2 /2a 2 ), (2.3) 
a(x, y) = a(x - y) = exp(-(x - y) 2 /2a 2 a ). 

and m(x,h)dh is the law of a Af(0, a 2 ) r.v. Y (centered Gaussian with variance a 2 ) condi- 
tioned on x + Y G X . 

The growth rate X(x) is maximal at x = and there is local competition between traits, 
in the sense that a(x,y) is maximal for x = y and is close to when \x — y\ is large. If 
the competition kernel was flat (a = 1), evolution would favor mutant traits with maximal 
growth rate. However, if competition is local, numerical simulations of the microscopic 
model give different patterns, as shown in Fig. 12.11 The pattern of Fig. 12.1( b). where the 
population, initially composed of traits concentrated around a single trait value, is driven 
by the evolutionary forces to states where the population is composed of two (or more) 
groups, concentrated around different trait values. This phenomenon is called evolutionary 
branching and has been observed in many biological models (see e.g. [26j [2U 02]) • It is 
believed to be a possible mechanism of traits separation that could lead to speciation [7]. 

In this particular model, the possibility of evolutionary branching seems to be governed by 
the values of a\, and a ai which represent respectively the width of the trait region with high 
growth rate and the interaction range. In Fig. 12.11 (a). a a > <j\, and there is no evolutionary 
branching, whereas in Fig. l2.lf b). a a < o\, and evolutionary branching occurs. We observe 
in both simulations that, in a first phase, the population trait support is concentrated 
around a mean trait value that converges to 0. In a second phase, new mutants feel two 
different selective pressures: high growth rate (traits close to 0) and competition (traits 
far from the rest of the population). If a a is small, the selection pressure is weaker for 
traits away from and allows the apparition of new branches. The goal of this article is 
to justify mathematically this heuristics. 

2.3 On scales 

In order to analyze the phenomenon of evolutionary branching, we are going to consider 
three biological asymptotics in the individual-based model: large population (K — > +oo), 
rare mutations (uk — ► 0) and small mutation amplitude. The combination of the two first 
scales will allow us to describe the polymorphic evolution sequence, we will focus on. This 
limit amounts to approximate the simulated dynamics of Fig. I2.1f a) and (b) of the previ- 
ous section by the one of Fig. I2.2f a) and (b), respectively. These scales and the biological 
heuristics of this approach were introduced in [26]. The main interest of the assumption of 
rare mutations is the separation between ecological and evolutionary time scales: the selec- 
tion process has sufficient time between two mutations to eliminate disadvantaged traits. 
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(a) p = 0.1, K = 1000, a = 0.01, a b = 0.9, a a = (b) p = 0.1, if = 1000, a = 0.01, cr 6 = 0.9, ct q = 
1.0. 0.7. 



Figure 2.1: Numerical simulations of the trait distribution (upper panels) and population size 
(lower panels) of the microscopic model with parameters (|2.3p . The initial population is composed 
of K individuals all with trait —1.0. 



Then evolution proceeds by a succession of phases of mutant invasion and phases of compe- 
tition between traits. We will choose parameters such that the ecological and evolutionary 
time scales are separated, leading to an evolutionary dynamics where competition phases 
are infinitesimal on the mutation time scale. In addition, the large population assumption 
allows one to assume a deterministic population dynamics between mutations, so that the 
outcome of the competition can be predicted. More formally, between two mutations, a 
finite number of traits are present, namely x\, . . . , x^, and the population dynamics can be 
reduced to a Markov process in N d . 

Assume that, for all i € {l,...,d}, -^{vq , ^-{ Xi }) nas bounded second-order moments 
and converge in distribution to nj(0) € M + . Then, as proved in [6] Thm.4.2], when K — > 
+oo, the process ^((^t , ls xi \), ■ ■ ■ , (f^, l{ x . d })) converges in distribution for the Skorohod 
topology to the solution of the d-dimensional competitive Lotka-Volterra system LV(d, x) 
with initial condition (ni(0), ... ,71^(0)). 

Definition 2.1 For any x = (x\, . . . ,Xd) € X 3 we denote by LV(d,x) the competitive 
Lotka-Volterra system defined by 

h(t) = F x (n(t)), 1 < i < d, t>0, (2.4) 
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(a) (i = 0.0001, K = 1000, a = 0.08, a b = 0.9, (b) (i = 0.0001, K = 1000, a = 0.08, a b = 0.9, 
a a — 1.0. a a = 0.7. 



Figure 2.2: Numerical simulations of the trait distribution (upper panels) and population size 
(lower panels) of the microscopic model with parameters (|2.3p . The initial population is composed 
of K individuals all with trait —1.0. The value of a is higher than in Fig. 12.11 so that the jumps 
are visible. 



where n(t) = (ni(i), . . . ,n d (t)), 

d 

Ff-(n) := UiGf (n) where Gf(n):=r(xi)-'^a(xi,Xj)nj. (2.5) 

i=i 

The equilibria of LV(d, x) are given by the intersection of hyperplanes (Pi)\<i<d, where 
Pi has equation either n« = or Gf (n) = 0. We need to introduce the following notion of 
coexisting traits. 

Definition 2.2 For any d>0, we say that Xi, ... ,Xd coexist if LV(d,x) admits a unique 
non-trivial equilibrium n(x) € (M^) d locally strongly stable, in the sense that the eigenval- 
ues of the Jacobian matrix of LV(d,x) at n(x) have all (strictly) negative real part. In 
particular, for all i £ {1, . . . , d}, 

Gf(n(x)) = and DF x (h(x)) = ((-a(x u x^x))^^. (2.6) 

In the monomorphic case (d = 1) and when r(x) > 0, the competitive Lotka-Volterra 
system LV(l,x) takes the form of the so-called logistic equation 

n x = n x (r(x) - a(x, x)n x ). (2.7) 
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The unique stable equilibrium of this equation is n(x) = r(x)/a(x,x). 

Similarly, in the dimorphic case where d = 2, the system LV(2, (x,y)) takes the form 

n x = n x (r(x) - a(x, x)n x - a(x, y)n y ) ^ 
hy = n y (r(y) - a(y, x)n x - a(y, y)n y ). 

Under Assumption (A2), the equilibria of (12. 8p are (0, 0), (n(x), 0), (0, n(y)) and possibly a 
non-trivial equilibrium in (M+) 2 . It is known (see e.g. [22]) that the non trivial equilibrium 
exists and is locally strongly stable, (traits x and y coexist), if and only if f(x; y) > and 
f(y; x) > 0, where 

f(y; x) = r(y) - a(y, x)n(x). (2.9) 

2.4 Convergence to the Polymorphic Evolution Sequence (PES) 

Our goal here is to examine the asymptotic behavior of the microscopic process when the 
population size grows to infinity as well as the mutation rate converges to 0, in a long time 
scale. Before stating our convergence result, we first give an idea of the argument used, 
extending the biological heuristics of [26] and the special case of the trait substitution 
sequence (TSS) developed in pfj (see also Section I3~T1) . 



2.4.1 Idea of the proof 

Let us roughly describe the successive steps of mutation, invasion and competition. The 
two steps of the invasion of a mutant in a given population are firstly the stabilization 
of the resident population before the mutation and secondly the invasion of the mutant 
population after the mutation. 

Fix r\ > 0. In the first step, assuming that d traits x\, . . . ,Xd that coexist are present, we 
prove that the population densities ((z/$ , ■ • • ; { v t ■> ^-{x d })) belong to the 77-neighborhood 

of n(x) with high probability for large K until the next mutant y appears. To this aim, 
we use large deviation results on the problem of exit from a domain [13] to prove that the 
time needed for the population densities to leave the 77-neighborhood of n(x) is bigger than 
exp(VK) for some V > with high probability. Therefore, until this exit time, the rate 
of mutation from trait Xi in the population is close to UKp(xi)X(xi)Kfii(x) and thus, the 
first mutation appears before this exit time if one assumes that 

1 «e VK . 



Kuk 

In particular, the mutation rate from trait Xi on the time scale t/Kux is close to 

p(xi)X(xi)ni(x.). 

In the second step, we divide the invasion of a given mutant trait y into 3 phases shown in 
Fig. 12.31 in a similar way as done classically by population geneticists dealing with selective 
sweeps [23] , 

In the first phase (between time and t\ in Fig. 12.31) . the number of mutant individuals is 
small, and the resident population stays close to its equilibrium density n(x). Therefore, 
the dynamics of the mutant individuals is close to a branching process with birth rate X(y) 
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Figure 2.3: The three steps of the invasion of a mutant trait y in a monomorphic population with 
trait x. 

and death rate /■*(?/) + Sf=i a (2A x i)^i( x )- Hence, the growth rate of this branching process 
is equal to the so-called fitness 

d 

/(y;x) = f(y,xi, ...,x d ) = r(y) -^2a(y,Xj)nj(x), (2.10) 

i=i 

describing the ability of the initially rare mutant trait y to invade the equilibrium resident 
population with traits x±, . . . , x&. If this fitness is positive (i.e. if the branching process is 
super-critical), the probability that the mutant population reaches density 77 > at some 
time t\ is close to the probability that the branching process reaches r]K, which is itself 
close to its survival probability [/(y;x)]+/A(y) when K is large. 

In the second phase (between time t\ and ti in Fig. 12. 3ft . we use the fact that, when 
K — > +oo, the population densities ((vf^ ,ls Xl \), . . . , (vf^ ,lr Xd \), (vf- ,ls y \)) are close to 
the solution of the Lotka-Volterra system LV(d+ 1, (aq , . . . , Xd, y)) with same initial condi- 
tion, on any time interval [0,T]. We will need an assumption (called (Bl) in Section 12,4.21) 
ensuring that, if rj is sufficiently small, then any solution to the Lotka-Volterra system 
starting in some neighborhood of (ni(x), . . . , n^(x), 0) converges to a new equilibrium 
n* e R d+1 as time goes to infinity. Therefore, the population densities reach with high 
probability the 77-neighborhood of n* at some time t^. 

Finally, in the last phase, we use the same idea as in the first phase: under the assumption 
(called (B2) in Section I2.4.2P that n* is a strongly locally stable equilibrium, we approx- 
imate the densities of the traits Xj such that n* = by branching processes which are 
sub-critical. Therefore, they reach in finite time and the process comes back to the first 
step until the next mutation. 

We will prove that the duration of these three phases is of order log-ff. Therefore, under 
the assumption 

log if < — , 

Kuk 
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the next mutation occurs after these three phases are completed with high probability. 



2.4.2 Assumptions 

As explained above, we need to introduce two assumptions on the Lotka-Volterra systems 
involved in the previous heuristics. These assumptions involve the fitness function defined 
in (|2.10p . This function is linked to Lotka-Volterra systems by the following property. 

Proposition 2.3 Assume that the traits x\, . . . , x d G X coexist. Then 

(i) For any i G {1, . . . , d}, /(xfjxi, . . . ,x d ) = 0. 

(ii) Iff(y;xi,...,Xd) < 0, the equilibrium (ni(x), . . . ,n d (x), 0) of LV(d+l, (x 1 , . . . , x d , y)) 

is locally strongly stable, and if f(y;xi, . . . ,x d ) > 0, this equilibrium is unstable. 

Proof The first point is immediate. The second point comes from the following relation 
between Jacobian matrices of Lotka-Volterra systems 



DF!"-w)(n 1 (x) 1 ...,S i (x) ) 0) 





—n\{x)a{xi 


y) 


DF x (n(x)) 








-n d (x)a(x d 


y) 


... 


/(y;x) 



Since x±, . . . ,x d coexist, all the eigenvalues of Di ?x (n(x)) have negative real parts. □ 



Let (B) denote the following Assumptions (Bl) and (B2). 

(Bl) Given any x = (x\, . . . , x d ) £ X d such that x±, . . . , x d coexist, for Lebesgue almost 
any mutant trait y G X such that f(y; x) > 0, there exists a neighborhood U C R rf+1 
of (n 1 (x), . . . , n rf (x), 0) such that all the solutions of LV(d + 1, (x\, . . . , x d , y)) with 
initial condition in U D converge as t —>■ +oo to a unique equilibrium in 

denoted by 

n*(xi, . . .,x d ,y). 

(B2) Writing for simplicity x d+ \ = y and n* for n*(xi, . . . ,x d +i), let 

J(n*) := {i G {1, ...,d+l} : n* > 0} and x* = (xf,i G I(n*)). 

Then, for Lebesgue almost any mutant trait x d+ \ as above, {x-i;i G ^(n*)} coexist 
and 

for all j £ J(n*) , /(a; i ;x*)<0. 

Assumption (Bl) prevents cycles or chaotic dynamics in the Lotka-Volterra systems. More- 
over, it also prevents situations as in Fig. 12.41 where the equilibrium n* is unstable. In 
this case, a solution of the Lotka-Volterra system LV(d + 1, (xi, . . . , x d , y)) starting from 
a point in any neighborhood of (n 1 (x), . . . , re d (x), 0), represented by the curved line in 
Fig. 12.41 does not need to converge to n*. 

Assumption (B2) is stated in the way permitting one to use the comparison with branching 
processes argument described in Section l2.4.1l when a mutant trait fixates in the population. 
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n 



(n 1 (x),...,n d (x),0) 

Figure 2.4: Assumption (Bl) prevents such situation. 

Definition 2.4 An equilibrium n of LV(d, (x\, . . . , x^j) is hyperbolic if the Jacobian ma- 
trix of LV(d, (xi, . . . , Xd)) at n has no eigenvalue with real part. 

Assumption (B2) can also be replaced by one of the following simpler two assumptions. 

(B3) For Lebesgue almost any mutant trait x^+i as in (Bl), n* is hyperbolic. 

(B4) For Lebesgue almost any mutant trait Xd+i as in (Bl), n* is strongly locally stable. 

Proposition 2.5 Assumptions (Bl) and (B2) are equivalent to Assumptions (Bl) and (B3), 
and to Assumptions (Bl) and (B4). 



Proof Let k 

I(n*) = {l,2,. 



Card(/(n*)). Assume that x\, . . . , Xd+i are reordered in a way such that 
, k}. Then it is clear, by the definition of coexistence and the fact that 



/ DF X * 






(—a(xi, Xj)n*)i<i<k, k+i<j<d+i 








f(x k+1 ,x*) 











V 






f(x d+1 ,x*) 



that (B2) implies (B4) which also trivially implies (B3). Assuming (B3), the stable mani- 
fold theorem (see e.g. [1 8j pp. 13-14) says that the set of points such that the solution of 
LV(d+ 1, (xi, . . . , Xd+i)) started at this point converges to n* is a submanifold of (M^) d+1 
of dimension I, where / is the number of eigenvalues of DF^ Xl ''"' Xd+1 \n*) with negative 
real part. In particular, if I < d+ 1, this manifold does not contain an open set of (M4.) , 
which is in contradiction with (Bl). Therefore, I = d + 1, which implies (B2). □ 

Therefore, Assumption (B2) essentially means that n* is hyperbolic, which is a property 
satisfied under very weak assumptions. In Section [31 various situations ensuring Assump- 
tions (Bl) and (B2) will be discussed. 
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2.4.3 Definition of the PES and Convergence Theorem 

Before stating our convergence result, let us first describe the limiting process (Z t ;t > 0) 
of the population process (vfj KuK ;t > 0) on the mutation time scale. This is a pure jump 
Markov process in A4q C M.f defined by 



M 



o ■- 



< ^2 n i{ x )dxi', d > 1, xi, . . . ,x n € X coexist > , 



which describes the successive population states at the evolutionary (mutation) time scale. 
As explained in Section 12.4.11 the quantity p(xj)X(xj)fij(x) is the re-scaled mutation rate 
in the resident sub-population with trait Xj and size %(x). When a mutant Xj + h is chosen 
with law m(xj,h)dh, the quantity ^^x(x-+lhf + ' 1S *he invasion probability of the mutant. 
Once the latter has invaded, the new population state is given by the asymptotic behavior 
of the Lotka-Volterra system described in Assumption (Bl). Because of the timescale 
separation (I2.13p . the stabilization of the population at its new equilibrium occurs before 
the next mutation and within infinitesimal time. 
Hence, the process Z will jump 

d d 

from ^nj(x)(5 Xi to y~]n*(xi, . . .,x d ,Xj + h)S x . + n* d+1 (x 1 , . . .,x d ,Xj + h)5 Xj+h 
i=l i=l 

with infinitesimal rate 

p(x j )A(x i )%(x)^^-^^m(x i , h)dh (2.11) 
for all j £ {1, . . . , d}. In other words, the infinitesimal generator of the process Z will be 

C A y^( x )<^) = / dh J2 p ( x i ) A ( X J ) "J ( x ) ~ T7 + ^ ' + m ( X 'J ' h ) x 
v t=i Jx j=i ^ + ' 

d d 

(p(^2n*(xi,...,x d ,Xj + h)5 Xi +n* d+1 (xi,...,x d ,Xj +h)5 Xj+h \ - tp( J^n^x)^ 



t=l i=l 



(2.12) 



We call this process Polymorphic Evolution Sequence (PES), by analogy with the so-called 
"Trait Substitution Sequence" (TSS) described in Section I3TT1 

Proposition 2.6 Under Assumptions (A) and (B), the PES is well-defined on W + and 
belongs almost surely to A4q for all time. 

Proof It follows from Assumption (A) and from (|2.6|) that the jump rates are bounded. 
Moreover, by Assumption (Bl), n*(xi, . . . , x n , y) is well-defined for almost all mutant traits 
y such that f(y; x) > 0, and by Assumption (B2), for such y, Yli=i n X ( x i> • • • > x d, y)$Xi + 
n* d+1 (x 1 ,...,x d ,y)6 y e Mq. □ 
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Theorem 2.7 Assume (A) and (B). Take x\,...,Xd € X that coexist and assume that 
Vq = Yli=i n f^Xi with nf — > n,(x) in probability for all 1 < i < d. Assume finally that 

Vy>0, logK < — — < exp(Vif). (2.13) 

T/ien, ( u t/K UK it > 0) converges to the process (Z t ;t > 0) wii/t infinitesimal genera- 
tor A2.12\) and with initial condition Zq = X]f=i ^«( x )^i' ^ e convergence holds in the 
sense of finite dimensional distributions on M.p equipped with the topology induced by the 
functions v \— > (v, f) with f bounded and measurable on X. 

The proof of this result follows closely the heuristic argument of Section 12.4.11 and is very 
similar to the proof of Theorem 1 of [4], that states a similar result in the case where no 
pair of traits can coexist. We detail in Appendix [A] all the steps and results of [4] that are 
modified in order to prove Theorem 12.71 

3 Particular cases and extensions of the PES 

In this section, we discuss various situations where Assumptions (Bl) and (B2) are satisfied 
allowing one to explicitly obtain the PES. 

3.1 The "no coexistence" case: an extension of the trait substitution 
sequence (TSS) 

In this section we characterize the case where the PES is well defined until the first co- 
existence time of two different traits. Assumption (B) with d = 1 (only one resident trait) 
involves the fitness function defined in (|2.9I) . 

Proposition 3.1 Let us assume the hypothesis 

(CI) For all x £ X, the set of y such that f(y;x) = has Lebesgue measure 0. 
Then (B) is satisfied for d = 1. 

Proof The assumption (B) for d = 1 involves 2-dimensional competitive Lotka-Volterra 
systems. Their asymptotic behavior is well-known (see e.g. [22j). In particular, 

• if f(x;y) > and f(y;x) < 0, any solution of LV(2, (x,y)) starting from R + x 
converges to (n(x),0), 

• if f(x;y) < and f(y;x) > 0, any solution of LV(2, (x,y)) starting from X M + 
converges to (0, n(y)), 

• if f(x;y) > and f(y;x) > 0, any solution of LV(2,(x,y)) starting from (M^) 2 
converges to n(x,y), 

• if f(x;y) < and f(y;x) < 0, (n(x),0) and (0, n(y)) are both locally strongly stable. 
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Moreover, all the equilibria are hyperbolic if and only if f(y,x) ^ and f(x;y) 7^ 0. 
Therefore, Assumption (CI) implies Assumption (B) for d = 1 since m(x, h)dh is absolutely 
continuous w.r.t. Lebesgue's measure. □ 

Let us now introduce the following killed PES {z[ l \t > 0) as a Markov jump process on 
M.oU{d}, where d is a cemetery state, with infinitesimal generator D$ defined as follows. 
Let v := n(x)5 x , then 

d$<p(n{x)8 x ) 

^p(n(x + h)6 x+ h) - <p(n(x)5 x f)p(x)\(x)n(x) ^w^'^ + 1 {f(x-,x+h)<o}™'{x, h)dh 
+ / (^(^)-^(w(K)^))p(x)A(x)n(»)l t/ ( x .^ +ft ) >0 j( x+ / l ; X )>o}m(x,/i)rf/i. (3.1) 



By construction, the killed PES {z[ l \t > 0) is always monomorphic before killing. Once 
the killed PES reaches the cemetery state d, it no longer jumps. 

This modification amounts to construct the killed PES as the PES, and send it to the 
cemetery state d as soon as a mutant trait y appears in a monomorphic population of trait 
x G X such that x and y coexist. Note that d is reached as soon as a mutant appears, 
that could coexist with the resident trait, even if this mutant actually does not invade the 
population. That explains why the invasion probability [f(y, x)]+/\(y) does not appear in 
the last line of (13.11) . 

The following proposition is a consequence of the previous discussion. 

Proposition 3.2 Under Assumptions (A) and (CI), the killed PES (Z^ ,t > 0) is almost 
surely well-defined and belongs almost surely to Mq U {d} for all time. 

The proof of the following result can be easily adapted from that of Theorem 12.71 

Corollary 3.3 With the same assumption and notation as in Theorem 12.71 except that 
Assumption (B) is replaced by Assumption (CI) and that d = 1, let 

tk := inf{t > : Supp{v^) = {x,y} such that (x,y) coexist}. 

Then the process 



(i^_l { _t_< Tjr} + d\ { _t_ >TKV t > 



converges as K — > +00 to the killed PES {z[ l \t > 0) with initial condition Z^ = n(x)5 x . 
The convergence is understood in the same sense as in Theorem 12.71 

Remark 3.4 The killed PES generalizes the so-called "Trait Substitution Sequence" (TSS), 
introduced in |26j . and rigorously studied in |4j. This TSS is obtained when the parameters 
of the microscopic model prevent the coexistence of two traits. Such an assumption, known 
as "Invasion-Implies-Fixation" (IIF) principle |15j is given by: 

(IIF) for all x G X , almost all y G X such that f(y; x) > satisfy f(x; y) < 0. 
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Hence, the TSS Z has on M+ the form 

Z t = n(X t )5 Xt , t>0, 
where X is a Markov pure jump process on X with infinitesimal generator 

L<p(x)= [ (ip(x + h)- f(x))p{x)X(x)n{x) H; ^ + m(x, h)dh. (3.2) 

J R i \{x + h) 

The killed PES {z[ l \t > 0) prevents the coexistence of two or more traits. Therefore, this 
process is not suited to our study of evolutionary branching in Section |H To this end, we 
need to examine a more general situation. 

3.2 The "no triple coexistence" case 

In this section we characterize the case where the PES is well defined until the first coex- 
istence time of three different traits. 

In the case d = 2 the fitness function (|2.10p of a mutant trait z in a population with two 
coexisting resident traits x and y is given by 

f(z;x,y) = r(z) - a(z,x)nx(x,y) - a(z,y)n 2 (x,y) (3.3) 

with 

- / x r(x)a(y,y) -r(y)a(x,y) 

ni{x,y) = — — r — -, 3.4 

a(x, x)a{y, y) - a(x, y)a{y, x) 

n 2 (x y) = r (^) Q; ( x ' x ) -r(x)a(y,x) 
2 ' a(x, x)a(y, y) - a(x, y)a(y, x) ' 

We need to extend this definition to any x,y € X such that f(x;y)f(y;x) > (and not 
only for the ones that coexist) . It can be easily checked that a(x, x)a(y , y) — a(x, y)a(y, x) 
cannot be under this condition. 

We can now introduce the following assumption : 

(C2) For all x,y € X that coexist, the set of z such that f(x;z) = 0, f(z;x) = 0, 
f(y;z) = 0, f(z;y) = 0, f(x;y,z) = or f(y;x,z) = (when these last quantities 
are defined) has Lebesgue measure 0. 

Proposition 3.5 There exists a set C coex (defined in (|3.6p ) such that Assumption (C2) 
implies (B) for d = 2 and for all (x, y, z) € X 3 \ C coex . 

Proof As in the previous section, we have to distinguish coexistence and non coexistence 
of three traits. To this aim we need to introduce the classification of the asymptotic 
behavior of 3-dimensional competitive Lotka-Volterra systems done by Zeeman [30]. Any 
3-dimensional competitive Lotka-Volterra system admits an invariant hypersurface S called 
carrying simplex, such that any non-zero solution of the system is asymptotic as t — » +oo 
to one in S (cf. [E]). S is a Lipschitz submanifold of homeomorphic to the unit 
simplex in by radial projection. Moreover, S is a global attractor for the dynamics in 
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\ {0} ([20], Thm.3]). In particular, one can deduce from the asymptotic behavior of 
trajectories on E the asymptotic behavior of trajectories starting in a neighborhood of S. 
Zeeman obtained a full classification of the topological equivalence classes of the 3-dimensional 
competitive Lotka-Volterra systems by determining the 33 topological equivalence classes of 
those systems restricted on their carrying simplex. (In an equivalence class, the trajectories 
of the systems are related by a homeomorphism of For a given system LV(3, (x, y, z)), 
the equivalence class to which it belongs is determined by the sign of the 2-dimensional fit- 
nesses /(x; y), f(y;x), f(x;z), f(z;x), f(y;z), f(z; y) and of the 3-dimensional fitnesses 
f(x;y,z), f(y;x,z), f(z;x,y) when they are defined. The equivalence classes of [30] are 
characterized by drawing on the unit simplex of the fixed points and the limit cycles 
of the system, joined by their stable and unstable manifold^]. 



Figure 3.1: The pattern on the carrying simplex that corresponds to the situation of Assump- 
tion (B). Traits x and y are the resident traits. 

The signs of the fitnesses correspond to the arrows in each diagram. For example, f(y; x) > 
means that, on the edge of the simplex that reach x and y, there is an arrow starting 
from x in the direction of y. In other words, the unstable manifold of (n(x),0, 0) contains 
(a part of) the edge of the simplex that reach x to y. Similarly, f(z; x,y) > means 
that f(x;y)f(y;x) > 0, i.e. that LV(3, (x, y, z)) has as fixed point {n\{x, y), n,2(x, y), 0), 
represented as the midpoint of the edge of the simplex linking x and y, and that this fixed 
point has an unstable manifold pointing in the direction of the interior of the simplex. The 
situation represented in Fig. 13.11 corresponds to this case, when x and y coexist. 
In order to check if Assumption (B) holds, we only need to restrict to the equivalence classes 
in which two traits coexist (the resident traits, say x and y), and the third (mutant) trait 
(say z) satisfy f(z;x,y) > 0. This situation corresponds to the cases where the carrying 
simplex has one side containing the pattern of Fig. 13.11 Among the 33 equivalence classes 
of [30], there are only 10 of them that satisfy this requirement, shown in Fig l3.21 We label 
them with the same numbers as in [30]. In Fig. 13.21 the figures obtained by exchanging x 
and y belong to the same equivalence class. An attracting fixed point of LV(3, (x,y,z)) 
is represented by a closed dot •, a repulsive fixed point by an empty dot o, a saddle point 
by the intersection of its stable and unstable manifolds. When the interior fixed point 
(the non-trivial equilibrium) is not a saddle point, it can be either stable or unstable. 
Depending on cases, this equilibrium can also be surrounded by one or several stable or 
unstable cycles. In particular, the sign of the fitnesses is not sufficient to determine the 
precise asymptotic behavior of the system near the interior equilibrium. The undetermined 

lr The stable manifold of an equilibrium is the set of starting points of the Lotka-Volterra system such 
that the solution converges to this equilibrium. The unstable manifold is defined in the same way, but for 
the time-reversed system. 
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type of these equilibria is represented in Fig. 13.21 by the symbol 0. 




Figure 3.2: The phase portrait on E for the 3-dimensional competitive Lotka-Volterra systems 
corresponding to the situation described in Assumption (B). 

Since no pattern as in Fig. 12.41 occurs in diagrams 7 to 12 in Fig. 13.21 we see that Assump- 
tion (Bl) is always satisfied except possibly in the cases of diagrams 26, 29, 31 and 33. It 
could be violated either if the interior equilibrium is surrounded by a stable cycle, or in 
diagram 26 in the case where the unstable manifold of the equilibrium (n\{x, y),fi2(x, y), 0) 
(midpoint of the lower edge of the simplex) admits the equilibrium (0,fii(y, z),fi2(y, z)) 
as limit point. Moreover, as before, all the steady states are hyperbolic if all the 2- and 
3-dimensional fitnesses are nonzero. 
Thus, if we define the set C coex as 

Ccoex '■= {(x,y,z) G X 3 : LV(3, (x,y, z)) belongs to classes 26, 29, 31 or 33}, (3.6) 

Assumption (B) will be satisfied for all (x,y,z) € X 3 \C CO ex as soon as Assumption (C2) 
is satisfied. 

Remark in addition that, as checked from Fig. 13.21 if x and y coexist and f(z;x,y) > 0, 
then (x, y, z) £ C coex if and only if both of the following properties are satisfied 

(PI) If f(y; x, z) is well-defined, then f(x; z), f(z; x) and f(y; x, z) have all the same sign. 

(P2) If f(x; y, z) is well-defined, then f(y; z), f(z; y) and f(x; y, z) have all the same sign. 

□ 

Assumptions (CI) and (C2) will be summarized in Assumption (C). 

Similarly as in Section EH we define the killed PES (Zf',t > 0) as a Markov pure jump 
process on AdoU{(9}, with infinitesimal generator £,( 2 \ The latter is given by (|2.12l) for d = 
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1, and for d = 2 and coexisting x\,X2, it is modified as follows. Let v := Y^=i n i{ x ii x 2)S Xi , 
then 

,.2 2 

C ( - i) cp(u)= / ^(^p(^n*(x 1 ,x 2 ,x j + h)5 Xt +nl(x 1 ,x 2 ,x j + h)5 X]+h ^J - (p(v)j x 
j=i i=i 

plxjjA^jrij^i,^) Xf^T^) 1 {(zi,*2,x J +h)?c coex }m{x j ,h)dh 

f 2 

+ / Y] {<p{d) ~ ^(^))p(^)A(x i )n i (xi,x 2 )l { ( sliX2iX , + / l)e c coe 4m-(x j ,/i)d/i. (3.7) 

This modification amounts to construct the killed PES as the PES, and send it to the 
cemetery state as soon as a mutant trait X3 appears in a dimorphic population of traits 
xi,X2 £ X such that the Lotka Volterra dynamics associated with traits xi,x%,X2 belongs 
to classes 26, 29, 31 or 33. Notice that the killed PES's support has at most two traits at 
each time. 

As in Section I3~T1 we deduce the following results. 

(2) 

Proposition 3.6 Under Assumptions (A) and (C), the killed PES [Z\ , t > 0) is almost 
surely well-defined and belongs almost surely to M.q U {d} for all time. 

Moreover this PES is the limiting process, on the mutation time scale, of the mutation- 
invasion process killed at the first triple-coexistence time. 

Corollary 3.7 With the same assumption and notation as in Theorem 12. 7\ except that 
Assumption (B) is replaced by Assumption (C) and that d G {1,2} ; let 

tk '■= inf{£ > : Supp{u^) = {x,y,z} such that (x,y,z) G C coex }. 

Then the process 

(u K t ! { ^ <fK} + d l { ^ >fK} ,t>0 

(2) (2) 

converges as K — » +00 to the killed PES (Z\ ,t > 0) with initial condition Zq = 

Note that the killed PES obtained in this section is sufficient to study the phenomenon of 
evolutionary branching in Section [4] when A'cR. 



4 Evolutionary branching and small jumps 

We will assume, in all what follows, that the initial population is monomorphic, in the 
sense that at time 0, all individuals have the same trait. 

We have seen in Section l3~T1 that, as long as there is no coexistence of two traits in the 
population (Assumption (IIF)), the support of the PES is reduced to a single trait and the 
asymptotic dynamics of the population is given by the killed PES Z {V > with generator lETTI) . 
In this section, our aim is to characterize the traits around which (IIF) fails and how 
evolutionary branching can occur in this CctSG, £IS observed in Fig. 12.2T b) . To do so, following 
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a general idea of the biological literature [261 El [111 [151 El [II], a key assumption is that the 
mutation amplitude is small. In this situation, we will study the behavior of the PES on 
large time scales which will allow us to observe a global evolutionary dynamics. 
In Subsection 14.11 we assume that (IIF) is always satisfied and we study the TSS with a 
small mutation step scaling e. We prove that on the longer time scale p-, the dynamics 
of the re-scaled TSS converges, when e tends to zero, to the solution of a (deterministic) 
ODE, called canonical equation of adaptive dynamics, or, more simply canonical equation. 
In Subsection 14.21 we come back to the general case. We show that (IIF) is satisfied on the 
time scale of the canonical equation and that evolutionary branching can only occur on a 
longer time scale. We are able to characterize the points, called "evolutionary singularities", 
in the neighborhood of which evolutionary branching may occur. In Subsection 14.31 we 
state and prove our main result of this section, giving a criterion for evolutionary branching 
in the limit of small mutational jumps. We thus rigorously prove a criterion stated with a 
heuristic justification in [26J. 



Let us firstly introduce the following additional technical Assumptions (A'): 

(A'l) The trait space X is convex. It is often implicitly assumed for biological models 
with continuous trait space. 

(A'2) The distribution m(x,h)dh has finite and bounded (in x) third-order moments. 

(A'3) The map x t— > m(x, h)dh is Lipschitz continuous from X to the set of probability 
measures P(M'), for the Wasserstein metric 

p(P 1 ,P 2 ) = inf { / \x-y\ R(dx,dy);R G V(R 1 xM!) with marginals P x and P 2 \. 
(A '4) The function 

g(y;x) = p(x)X(x)n(x) ' 

is continuous on X 2 and of class C 1 with respect to its first coordinate, where /(•;•) 
is defined in (|2.9|) . Since X is a compact set of M. 1 , there exists a constant 7 > such 
thatVx,yG<Y, \g(y; x)]+ < 7. 

Later in this section, we will also need Assumption (A"): 

(A") The functions X(x) and fj,(x) are C 3 on X and the function a(x,y) is C 4 on X 2 . 
Note that (A") implies (A'4). 

Finally, let us introduce the parameter e G (0, 1] scaling the size of mutation. Since X is 
convex, x + eY G X a.s. for all x G X and < e < 1, where Y is distributed following 
m(x, h)dh. Therefore, it is possible to define a PES in which mutational jumps are scaled 
by the parameter e, by replacing in its generator (|2.12l) m(xj, h)dh by m(xj, h)dh o H~ l 
for all j G {1, . . . , d}, where H £ (h) = eh. Under Assumptions (A) and (B), we define this 
way a "rescaled PES" (Zf,t > 0). If only Assumptions (A) and (C) are satisfied, we do 
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similar changes in (13771) to obtain a "rescaled killed PES" (zf ),£ ,t > 0). Finally, we do a 
time scaling of order 1/e 2 to obtain the rescaled PES 

^ e J Z^ e 2 if Assumptions (A) and (B) are satisfied 



* \^t/l^ if on ly Assumptions (A) and (C) are satisfied. 

Since both Zf and zf^' e agree as long as there is no triple coexistence, and since we will 
only be interested in the sequel to the cases where the PES is monomorphic or dimorphic, 
we will not need to distinguish between these two cases. 

4.1 The TSS and the Canonical Equation of Adaptive Dynamics 

Doing a similar time scaling as for Z e , we can define for all e G (0, 1], the e-rescaled TSS 
(Xf,t > 0) by modifying the generator (13. 2|) as follows. For all C^-valued function tp, 

L 6 ip(x) = 4r / (ip(x + eh) — tp(x))[g(x + eh; x)] + m(x, h)dh. (4.1) 
£ JW 

From a mathematical point of view, the multiplicative term e~ 2 takes into account that 
the integral term is of order e 2 , because of g(x;x) = and Assumption (A'4). 

Let us now state the convergence theorem of the rescaled TSS to the canonical equation 
of adaptive dynamics. Its proof is based on a standard uniqueness-compactness method. 

Theorem 4.1 Assume (A) and (A'). Suppose also that the family of initial states {Aq}o< £ <i 
is bounded in L 2 and converges in law to a random variable Xq as e — > 0. 

Then for each T > 0, the sequence (X £ ) converges when e — > 0, for the Skorohod topology of 
D([0, T], X), to the process (x(t),t < T) with initial state Xq and with deterministic sample 
paths, unique solution of the ordinary differential equation, known as canonical equation of 
adaptive dynamics. 

dx f 

— = h[h ■ Vig(x;x)] + m(x,h)dh. (4.2) 
at M 

Remark 4.2 In the case where m(y, •) is a symmetrical measure on M. 1 for all y G X , 
Equation (14.21) gets the classical form, heuristically introduced in [8], 

^ = l K (x)V ig (x;x), (4.3) 
where K(x) = (kij(x)) 1<:i - <f is the covariance matrix of m(x, h)dh. 
Proof 

(i) Uniqueness of the solution of Equation (|4.2p with given initial condition. 

Let us show that a(x) = f Rl h[h ■ Vig(x; x)] + m(x, h)dh is Lipschitz continuous on X. We 
have 



a(x) — a(x')\\ < / \\h\\ x \[h ■ V\g(x;x)] + — [h ■ Vig(x';x')] + \ m(x,h)dh 



+ 



h[h ■ Vxg(x'\ x')]+(m(x, h) — m(x', h))dh 



(4.4) 
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Because of \[a] + — [b] + \ < \a — b\, Assumptions (A'2) and (A'4), and that the support of 
all measures m(x,h)dh is included in a bounded set, the first term of the right hand side 
of (|4.4I) is bounded by some constant times \\x — x'\\. 

If we denote by £ the vector V\g(x'; x') and tp(h) = h[h ■ then 

\Mh)-W)\\ < \\(h-h')[h^] + \\ + \\h'({h^] + -[h'^} + )\\ <2U\\ \\h-h'\\ (\\h\\ + \\h'\\). 

Thus, using the dual form of the Kantorovich-Rubinstein metric (see Rachev [28]) and 
(A'3), one obtains that the second term of the right-hand side of (|4.4p is also bounded 
by some constant times \\x — x'\\. Hence Cauchy-Lipschitz Theorem can be applied and 
(x(t),t > 0) is uniquely defined. 

(ii) The processes X £ , e > 0, with generator L e can be constructed on the same 
probability space. 

Recall the definition of 7 in Assumption (A'4). 

Lemma 4.3 Assume (A) and (A'). Let P) be a probability space and N(dh,d6,ds) 

be a point Poisson measure on M. 1 x [0, 1] x R + with intensity , yfh(h)dhd9ds. Let e > 
and denote by N £ the image measure of N by the mapping s 1— > e 2 s. Let X s . be a X -valued 
random variable, independent of N. Then the process X s defined by 

Xf=X%+ (eh) lr lg(x e +eh . x s )]+m{x e hn N E {dh,de,ds), (4.5) 

JR 1 xf0.llxf0.il i °< = ^TTH — f 



! x[0,l]x[0,t] | y < ffjjY 

is a jump Markov process with generator L £ . Its law will be called P^e . 
Indeed, using Ito's formula, one observes that for a bounded function <p on X, 

<p(X!) - V9(A £ ) 

- / / [p{X$ B + eh) - (p(X e £2s )) l {£ 2 s < t} g(XI 2s + eh; A| 2s )m(A| 2s , h)dhddds 

JO JR l x[0,l] 
is a martingale, which implies the result, 
(hi) Tightness of the sequence of laws {P £ X e} £ >o- 

We will use the Aldous criterion [I]. Let r be a stopping time less than T and (5 e ) positive 
numbers converging to when e — > 0. We remark that \g(x + eh;x)\ < eC\\h\\, by an 
expansion of g with respect to its first variable and the fact that g(x; x) = 0, and since 
Vi<7 is bounded by a constant C. We have 



E( 



X £ t+Se - A T 1) = E (£ +5s J \\eh\\\g(Xg_ + eh; X £ s _)} + m(Xl_,h)dh^ < CM 2 5 £ , 



where M2 = f \\h\\ 2 m(h)dh. Then, for any a > 0, 

p(re+«5 E - k 11 > «) < ^-^^ when e °- 

This gives the first part of the Aldous criterion. For the second part, we have to prove 
the uniform tightness of the laws of (sup t<T ||Xf||) e >o. We use Ito's formula to write 
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e\\2\ 



< 



{Xf) from (|4.5|) . Schwarz' and Doob's inequalities and obtain that E(sup t<T 
Ct(E(||Xq || 2 ) + 1), where Ct is a constant depending on time T, on Mi and on an 
upper-bound of [<?]+. Since (Aq) < £ <i is bounded in L 2 , the tightness of the laws of 
(sup t<T ||X t e ||) e> o follows. 

(iv) Convergence of the generators. 

Let us now prove that 



VV G C 2 (X), -^L e ip -> L°v? uniformly on X, 



(4.6) 



where L £ is defined in (|3.2I) and L° is defined by 



LV(x) 



(/i • V(^(x))[/i • V\g(x; x)] + m(x, h)dh, 



where Vcp(x) is the gradient vector of <p(x). We have, 



-zL e <p(x) - L°<p(x) 



+ 



ip{x + eh) — <p(x) 



< / [h-Vig(x;x)]+x 



g(x + eh; x) 



(p(x + eh) — (p(x) 



h ■ Vip{x) 



m(x, h)dh 



[h ■ Vxg(x;x)x)\. 



m(x,h)dh. (4.7) 



Let us call I\ and I2 the quantities inside the integral in the first and the second term, 
respectively. Now, (p is C , g(x;x) = and by Assumption (A'), g(x;y) is C 1 with respect 
to the first variable x. So, we can find #1, 62 and #3 in [0, 1] depending on x and h such 
that 

h = [h ■ Vig(x; x)] + x \h ■ V(p(x + 6 3 eh) — h ■ V(f(x)\; 

I2 = \h ■ Vlp(x + 0\eh)\ x \[h ■ Vig(x + 02eh; x)] + — [h ■ V\g(x; x)} + \. 

Since ip is in C 2 , and because of Assumption (A'), we can choose a number C such that 
V(p and V\g are both C-Lipschitz and bounded by C on X and X 2 respectively. Then 

h < C\\h\\ x ||/t||C||0 3 ^|| < eC 2 ||/i|| 3 ; 



I 2 < C\\h\\ x \h ■ V\g{x + 9 2 eh,x) - h ■ Vxg(x,x)\ < eC 2 \ 
It remains to put these bounds in Equation (14.71) to obtain: 



1 



L £ ip(x) - L°ip(x) 



< 2eC 2 [ \\hfm(x,h)dh. 
Jr 1 



We conclude using Assumption (A'2). 

(v) Martingale problem for limiting distributions. 

Finally, let us show that any accumulation point P of the family of laws {Pxgl 011 
D([0, T],X) is the law of the process X solution to (|4.2p with initial state Xq. Fix such 
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a P. Let us endow the space D([0, T],X) with the canonical filtration Q t , and for any 
ip G C 2 (X), let us define on this space the processes 

Mf(w) = <p(w t ) - <p(w ) - [ L°(p(w s )ds 

1 1 



o 



Ml ,lp {w) = tp(w t ) - <p(wo) - / -^L £ ip(w s )ds. 

Jo e 

We will show that M v = P-a.s. Fix tp G C 2 {X). It is standard, using Ito formula for 
jump processes, to show that, under P^e, M £,lf is a square-integrable ^-martingale and 
that 



M^(X*) = [ { V (Xl + eh) - V {X<.)) 

JK ! x[0,l]x[0,I] 



1 ( lg{X*_+eh,xe_)] + m(Xg ft ) ~| ^ (dk,d6,ds) 

7 fh(h) 



where iV e = iV e — q 6 is the compensated Poisson measure associated with N 6 , and 
q £ (dh,d9,ds) is the image measure of jrh(h)dhd9ds by s i— ► e 2 s. Thus, using computation 
similar to (14,11) . 

E £ ((M^> 4 ) = ^E £ ( J* J MX 6 . + eh) - V {Xt)f[g{Xl + eKXt)} + m(X!,h)dhds 

<CC'M 3 te, (4.8) 

where E £ denotes the expectation under P^e , C is a bound for Vip, and M3 a bound of 
the third-order moment of m(y,h)dh. Using (|4.8|l and the fact that Mf(w) = M^'' fi (w) + 
f (-^L £ ip(w s ) — L°(p(w s ))ds, it follows that 

E £ (|Mf | 2 ) < 2t 2 ||^ LV - LVllL + 2C 2 C ,2 M 3 W 

which converges to when e — > thanks to (|4.6p . Moreover by (14.51) . we have that almost 
surely, sup 4<T \\X £ — -X" £ _| < C"e, which implies that each limit process X with law P is 
almost surely continuous. Hence, for any t G [0, T], the functional to 1— > (p{ w t) — <p{wq) — 
L L°<p(w s )ds is continuous at X for the weak topology and since P is the weak limit of 
an extracted sequence of (P £ X e), it follows that, under P, M v [w) = a.s, which concludes 
the proof. □ 



4.2 PES and Evolutionary Singularities 

Until the end of Section 31 we will assume for simplicity that the trait space is 
one-dimensional (I = 1), i.e. X C R. 

We have proved in the last subsection that, when e — > 0, the TSS is very close to the 
solution of the canonical equation (|4.2p on any time interval [0, T], The equilibria of this 
equation are given by the points x* such that either d\g(x*;x*) = 0, or J R m(x* , h)dh = 
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and d\g{x*;x*) > 0, or J R _ m(x*,h)dh = and dig(x*;x*) < 0. We will concentrate on 
the points such that d\g{x*;x*) = 0, or equivalently, d\f (x*;x*) = 0, since 

dig(x;x) = dif(x;x)p(x)X(x)n(x) = p(x)n(x)dif(x; x). 
X(x) 

Remark that, since f(x; x) = for all 

d 1 f{x;x) + d 2 f{x;x) = 0, Vx G X (4.9) 
9n/(x;x) +25i 2 /(x;x) +a 22 /(x;x) = 0, Vx G Af. (4.10) 

Therefore, 5i/(x*;x*) = <9 2 /(x*;x*) = 0. 



Definition 4.4 Points x* such that d\g{x*;x*) = 0, or equivalently, d\f{x*;x*) = 
d2f(x*;x*) = are called evolutionary singularities (ES). 

Lemma 4.5 Assume (A), (A') and (A"). 

(1) The solution x(t) of (14.21) starting from a point that is not an ES cannot attain an ES 

in finite time. 

(2) Assume that x(0) is not an ES and let It = {x(t),t £ [0, T]}. Then, for any sufficiently 

small rj > 0, for any x at a distance to It smaller than rj and for any y sufficiently 
close to x, x and y satisfy (IIF) and (y — x)f{y;x) has constant sign. 

Proof (1) Let c be a constant such that x h- > L h[h ■ d\g{x; x)] + m(x, h)dh is c-Lipschitz 
(the fact that this is a Lipschitz function is shown in the proof of Theorem 14. Then, for 
any ES x*, 

-(x(t) - x*) 2 < 2 \±(t)\ \x(t) -x*\<2c (x(t) - x*) 2 . 
dt 

Thus, \x(t) -x*\> \x{0) - x*\ exp(-ci) > 0. 

(2) Remark first that, from Point (1), C = inf xe / T \d\f{x{t); x(t))\ > 0. Therefore, for 
i] > sufficiently small, {x G X : dist(x,I T ) < i]} C {x G X : |<9i/(aj(t); x{t))\ > C/2}. 
Fix such an rj. 

Let us now consider some point x in X such that d\f{x;x) > C/2. Consider first y in X 
such that x < y. Using that f(x;x) = and (|4.9p . a second-order expansion of f(y;x) at 
(x,x) implies that f(y;x) > C(y — x)/4 provided that \y — x\ < ^y, where C > is a 
constant uniformly upper-bounding the second-order derivatives of /(•; •) on the compact 
set X 2 . Under the same condition, f(x;y) < C(x — y)/4. Therefore, f(x;y)f(y;x) < if 
\y — x\ is small enough and (y — x)f(y; x) has constant sign. This reasoning gives the same 
conclusion if y < x or d\f(x;x) < —C/2, giving the required result. □ 



Now we come back to the rescaled PES (Zf,t > 0) defined in the beginning of this section 
and assume that its initial condition Z$ is monomorphic. We want to determine when 
evolutionary branching can occur in this process. This requires that (IIF) (ensuring non 
coexistence) fails. For e > 0, we define the first coexistence time 

r e = m£{t > 0, f(Z%, ; Z\_) > and f(Zf_; Zf) > 0}, 
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and for any r\ > 0, the entrance time of the process in a 77-neighborhood of an ES x* , 

9 £ v = inf{i > 0, Supp(Zf) n (x* - 77, a;* + 77) ^ 0}. (4.11) 

Theorem 4.6 Assume that (A), (A'), (A") and (B) or (C) hold. Assume also that Zq = 
n(x)5 x where x G X is not an evolutionary singularity. Then, 

(i) For any T > 0, 

limP(T e > T) = 1. 

e— +0 

Moreover, for all rj > 0, 

limP(Vt G [0,21, Card(SuppfZf)) = 1, |ISupp(Z|) - x(t)\\ < rf) = 1. 

e— >0 

(ii) For any r\ > 0, there exists £o > such that, for all e < Eq, 

P(6^ < t £ ) = 1 and 
P(Vt G [0,6^], Supp(Zf) = {y/} witt t i ^ y/ monoionoMs on [0,6^]) = 1. (4.12) 

Proof (i) Before the stopping time r e , and since the initial condition is monomorphic, 
it is clear that the support of Zf is a singleton whose dynamics is that of the rescaled 
TSS (Xf,t > 0). Because of Theorem 14.11 when e — > 0, the TSS is close to the canonical 
equation. In particular, for all rj > 0, its values on the time interval [0, T] belong to the 
set {x G X : dist(x,2r) < v} with probability converging to 1. Moreover, since X is 
compact, Supp(m(a;, •)) C X — x is included in the closed ball of PJ centered at with 
diameter 2diam(^f). Therefore, the distance between a mutant trait and the trait of its 
progenitor in the rescaled PES Z e is a.s. less that ec, where c is a constant. Hence, the 
result immediately follows from Lemma [4.51 

(ii) We also deduce from this lemma that for any T > such that It H (x* — 3r//2, x* + 
377/2) = 0, lim e _»o > T) = 1. Moreover, the process Y"/ in (|4.12l) . which is exactly the 
TSS of the previous section, is almost surely monotonous before time □ 

Remark 4.7 Theorem \4-6\ implies that, when the initial population is monomorphic and 
away from evolutionary singularities, evolutionary branching can only occur in the neigh- 
borhood of an evolutionary singularity and on a longer time scale than T/e 2 when e — > 0, 
for all T > 0. 

The next result shows that we can restrict to ES that are not repulsive for the canonical 
equation. 

Proposition 4.8 Under the assumptions of Theorem 14. 6^ coexistence of two traits can only 
occur in the neighborhood of evolutionary singularities x* G X which are not repulsive, i. e. 
which satisfy 

d 2 2f(x*;x*)>duf{x*;x*). (4.13) 

More precisely, for any neighborhood U of the set of evolutionary singularities satisfy- 
ing (|4.13|) . for all £ small enough, 

P(r £ < +00 and Supp(Z£_) g U) = 0. 
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Proof Let us remark that an ES such that 

d n f(x*; x*) + d 12 f(x*; x*) > 0. (4.14) 

is always a repulsive point for the canonical equation, in the sense that, for any solution 
x(t) of the canonical equation starting sufficiently close from £*, the distance between x(t) 
and x* is non-decreasing in the neighborhood of time 0. In other words, there exists a 
neighborhood U of x* such that no solution of the canonical equation starting out of U can 
enter U. To this end, we remark that (I4,14p implies that there exists r] x * with 

• dig(x;x)>0 if x G (x*,x* + r) x *], 

• dig(x;x)<0 if x G [x* — r] x * , x*), 
and conclude in view of (14,21) , 

Observe that, by (l4~TO . (ITOH is equivalent to d n f(x*;x*) - d 22 f(x*;x*) > 0. 
Let S be the set of repulsive ES and define V = L) x *^s(x* — r] x *,x* — rj x *). Fix U as in 
the statement of Proposition 14.81 and assume (without loss of generality) that U n V = 
and x U U V. Let [a, b] be any connected component of X \ (U U V). Since dif(y, y) ^ 
for all y G [a,b], reproducing the argument of the proof of Theorem 14.61 easily shows 
that coexistence never happens in a monomorphic population with trait in X \ (U U V) 
if e is sufficiently small. Similarly, for e sufficiently small, no mutant in V born from a 
monomorphic population with trait not belonging to V has a positive fitness. Therefore, 
the TSS cannot drive the population inside V starting from outside. Thus Proposition 14.81 
is clear. □ 

4.3 Evolutionary branching criterion 

In this section we will prove a criterion of evolutionary branching. We need the following 
last assumption. 

(A'") For any x in the interior of X, J R m(x, h)dh > and J* R m(x, h)dh > 0. 
4.3.1 Definition and main result 

We first need to precisely define what we mean by evolutionary branching. 

Definition 4.9 Let x* be an ES. For all r\ > 0, we call rj-branching the event 

• there exists t\ > such that the support of the PES at time t± is composed of a single 
point belonging to [x* — r], x* + rj\ 

• there exists t 2 > t\ such that the support of the PES at time t 2 is composed of exactly 
2 points distant of more than rj/2 

• between t\ and t 2 , the support of the PES is always a subset of [x* — n, x* + rj] ; and 
is always composed of at most 2 traits, and has increasing (in time) diameter. 
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We only consider binary evolutionary branching. We will actually prove that the simulta- 
neous subdivision of a single branch into three branches (or more) is a.s. impossible. Note 
that this notion of evolutionary branching requires the coexistence of two traits, but also 
that these two traits diverge from one another. 

Our main result is the following. 

Theorem 4.10 Assume (A), (A'), (A"), (A'") and either (B) or (C). Assume also that 
Zq = n(x)5 x and that the canonical equation with initial condition x converges to an ES 
x* in the interior of X such that 

d 2 2f(x*;x*)>d n f(x*;x*) (4.15) 
and d 22 f{x*;x*) + d n f(x*;x*)^0. (4.16) 

Then, for all sufficiently small r\, there exists eo > such that for all e < Eq, 

(a) if dxif(x*;x*) > 0, ¥ £ (77 -branching) = 1. 

(b) if dnf(x*;x*) < 0, P £ (r]-branching) = 0. Moreover, 

P e (Vi > 9 £ n , Card(Supp(Zf)) < 2 and Supp(Zf) C (x* - 77, x* + r?) ) = 1, 
where 6^ has been defined in (|4.11|) . 

This criterion appeared for the first time in [26] with an heuristic justification. We see that, 
locally around x*, one of the two following events can occur almost surely: either there is 
binary evolutionary branching and the two branches diverge monotonously, or there is no 
evolutionary branching, and the population stays forever inside any neighborhood of x*. 
Coexistence can occur in this case, but cannot drive the support of the population away 
from a small neighborhood of x* . We will actually prove that, in this case, as soon as 
there is coexistence of two traits in the population, the diameter of the support of the PES 
can only decrease until it reaches (i.e. until the next time when the population becomes 
monomorphic) . 

We give in the following subsections a full proof of this result. In Section l4.3.3l we will prove 
regularity results on the 2- and 3-dimensional fitness functions and give their second order 
expansions in the neighborhood of evolutionary singularities. A first corollary of this result 
is given in Section 14.3.41 where, using the results of M.-L. Zeeman [30J and Fig. 13.21 we will 
show that no triple coexistence can occur in the neighborhood of evolutionary singularities. 
Finally, a case by case study of the zone of coexistence and of the signs of fitness functions 
in the neighborhood of an evolutionary singularity will allow us to conclude the proof in 
Section 14.3.51 

Before coming to the proof and in order to illustrate the difference between coexistence 
and evolutionary branching, we state a result that will be needed in the course of the proof 
of Theorem 14.101 Its proof will be given in Subsection 4.3.4. We recall that two traits x 
and y coexist if and only if f(x; y) > and f(y; x) > 0. 

Proposition 4.11 Assume (A) and that X, \i and a are C 2 . Let x* £ X be any ES. 
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(a) If On f(x*;x*)+ 822/ (x*;x*) > 0, then for all neighborhood U ofx*, there exist x,y G U 

that coexist. 

(b) If d\if(x*;x*) + d22f(x*; x*) < 0, then there exists a neighborhood U ofx* such that 

any x,y ElA do not coexist. 

This shows that the criterion of evolutionary branching (dnf(x*;x*) > 0) is different 
from the criterion of coexistence (dnf(x*;x*) + d22f{x*;x*) > 0). In particular, if one 
assumes as in Theorem 14.101 that d22f(x*;x*) > dnf(x*;x*), the evolutionary branching 
condition dnf(x*;x*) > implies the coexistence criterion dnf(x*; x*) + 822 f(x*; x*) > 0, 
as expected. 



4.3.2 Example 

Let us come back to the example introduced in Subsection 2.2. 
The fitness function is 

f{y- x) = \(y) - a(y, x)n(x) 

y 2 \ ( (x — y) 2 \ ( x 2 



Computation gives 



-Pl-^-exp^ ^jexp v 2(j , 



d x f{x*- x*) = - X - exp ( - i-4- j = ^ x* = 0. 

Moreover, dnf(0; 0) = \ — \ and ^22/(0; 0) = -3- + -\. Thus, the coexistence criterion 

of Proposition |4~TT1 (a) is always satisfied. We furthermore observe that (|4.15l) and (|4.16l) 
hold, and that 

dn/(0;0) >0^a a <a b . 

Then if a a < we have almost surely branching and when a a > we have only 
coexistence. This is consistent with Fig. 12.21 (a) and (b). 



4.3.3 Trait smoothness of fitnesses around evolutionary singularities 

The problem of local expansion of fitness functions has been already studied in [TO] for 
general models. In this section, we establish regularity and expansion results on our 2- and 
3-dimensional fitness functions in the neighborhood of evolutionary singularities. To this 
aim, we need the following lemma. 

Lemma 4.12 Let h(x, y, z) be a C k function for k > 1 defined on X 3 such that h(x, x, z) = 
for all x,z E X . Then, the function 

[x,y,z) h ' 



x-y 



can be extended on {x = y} as a C k 1 function h(x, y, z) on X 3 by setting h(x, x, z) 
d\h{x, x, z) for all x, z 6 X. 
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Proof Taylor's formula with integral remainder yields 

h(x,y,z) ''' 



x-y Jo 



dih(y + (x- y)u,y,z)du 



for all x ^ y. The right-hand side also has a sense for x = y and defines a C k 1 function 
on X 3 . □ 



Let x* G X be an ES as in the statement of Theorem 14. 101 By Assumptions (A) and (A"), 
the 2-dimensional fitness function f(y;x) defined in (12,91) is well-defined for all x,y G X 
and is a C 3 function. We extend the definition of the 3-dimensional fitness function 

f(z;x,y) = r(z) - a(z, x)ni(x, y) - a(z,y)n 2 (x,y), 

where nj(x,y), i = 1,2, are defined in (13. 4p and (13. 5p to all x,y £ X such that 

a(x, x)a(y, y) - a{x, y)a(y, x) ^ 0. 

We will also use the notation 

a = d n f(x*;x*) and c = d 22 /(x*; x*). (4.17) 

Note that, by ([4TTU1) . 

d 12 f(x*;x*) = -^. (4.18) 

Proposition 4.13 Under the assumptions of Theorem I4.10j. i/ie following properties hold. 

(i) For all x,y € X in a neighborhood of x* , 

x^y =^ a(x, x)a(y, y) ^ a(x, y)a(y, x). 
This implies in particular that n(x, y) and f(-;x,y) are well-defined for such x,y. 

(ii) When x, y — > x* in such a way that x 7^ y, and for all z G X , 

v( x* 1 

ni(x,y) +n 2 (x,y) — > n(x*) = — - — — ; (4.19) 

a{x*, x*) 

f(z;x,y) — » f(z;x*). (4.20) 
(hi) With the notation (|4.17l) . as x,y — > x*, 

f{y,x) = -{x - y)(c(x - x*) - a{y - x*)) + o(\x - y\ (\x - x*\ + \y - x*\)). (4.21) 

(iv) The function f(z; x, y) can be extended as a C 2 function on {(x, y,z) : z G X,x,y £U} 
for some neighborhood U ofx* in X . Still denoting by f(z;x,y) the extended function, 
as x, y — > x* , 

f(z;x,y) = -(z - x)(z - y) + o(\z -x\\z - y\). (4.22) 
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Proof Let D(x,y) := a(x, x)a(y,y) — a(x,y)a(y,x). It follows from Lemma [4. 121 that 
D(x,y)/(x — y) can be extended on X 2 as a C 3 function, which has value 

dia(x, x)a(x, x) + d2(x(x, x)a(x, x) — d\a{x, x)a(x, x) — a(x, x)d2a(x, x) = 

at the point (x,x). Therefore, Lemma 14,121 can be applied once more to prove that 
D(x,y)/(x — y) 2 can be extended as a C 2 function D(x,y) on X 2 . Hence, an elemen- 
tary computation involving the second-order Taylor expansion of D(x, y) yields that 

D(x,y) = (x — y) 2 [a(x* , x*)d\2a(x* , x*) — d\a(x* , x*)d20t(x* , x*)) + o(\x — y\ 2 ). 

Thus, Point (i) follows from the fact that a(x* , x*)d\20i{x* , x*) ^ d\a{x* , x*)d2a(x*, x*), 
which is a consequence of (|4.16|) . Indeed, an elementary computation shows that 

w/ * A / *,dna(x*,x*) 



a = r (x*) — r{x*)- 



a(x*,x*) 



and c=-r"(x*) + 2r'(x^ dia( - X * ,X * ) 



+ r(x*) 



a(x*,x*) 

a(x*,x*)(d u a(x*,x*) + 2di 2 a(x* , x*)) - 2dia(x*,x*) (dia(x*,x*) + d 2 a(x* , x*)) 



a(x*,x*) 2 
Using the fact that 

r (x ) = r(x ) — -. r— (4.23) 

since x* is an ES, we have that 

a 2 (x* ,x*)(a + c) = 2r(x*)(a(x* , x*)di2a(x* , x*) — d\a(x* , x*)d2a(x* , x*)) . 

Hence, 

a(x* , x*)d\20i{x* , x*) — dia(x* , x*)d2a(x*, x*) ^ -<=>■ a + c ^ 0. 

In particular, this implies that the function D(x,y) is non-zero in a neighborhood of x*. 
For Point (ii), observe that 



ni(x,y) + n 2 (x,y) 



r ( x \ a (y,y)- a (y, x ) . r ( v \ a ( x , x )- a ( x ,y) 

V ) x—y ' x—y 

(x - y)D(x,y) 



By the proof of Lemma [4.121 the numerator can be extended as a C 3 function h(x,y) by 
setting 

h{x,y) = -r(x) / d 2 a(y,y + (x - y)u)du + r(y) I <9 2 a(a;, y + (x - y)u)du 
Jo Jo 

for all x,y G X. In particular, h(x,x) = for all x € X. Therefore, Lemma [4. 121 can be 
applied once more to prove that ni(x,y) + n2(x,y) can be extended as a C 2 function in 
the neighborhood of x* and that 



x,y-*-x*,x^y ' ' D(x*,x*) a(x* , x*)di 2 a(x* , x*) - d\a(x* , x*)d 2 a(x* , x*) 
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Hence, (ETT9D and then (l4~20ll follow from (l4~23l) . 

Point (iii) is obtained from the fact that f(x; x) = 0, from Lemma 14,121 and from the 
second-order Taylor expansion of f(y;x). In this computation, one must use the fact that 
x* is an ES and (14T81) . 

The fact that f(z;x,y) is C 2 in hi x hi x X can be proven exactly as the regularity of 
n\{x,y) +n,2(x,y) above, observing that 

r(x) "(^'^"(^""(^"fa'^) _|- r(y) a ( z >y} a ( x > x }~ a ( z > x } a ( x 'y"> 

f(z;x,y) =r(z) — v 



(x - y)D(x,y) 

Therefore, using the fact that f(x;x,y) = f(y;x,y) = 0, Lemma f4. 121 can be applied twice 
to prove that 

1 

f{z;x,y) = -{z - x){z - y) + o(\z - x\ \z - y\) 

for some constant 7 G M. The second-order Taylor expansion of f(z;x,y) shows that 
7 = d u f{x*;x*,x*). Now, because of KM . d u f(z;x*,x*) = d u f(z;x*) for all z € X. 
Hence 7 = a, which ends the proof of Point (iv). □ 



Remark 4.14 Let us remark that, if x* is not an evolutionary singularity, Point (ii) of 
Proposition 14,131 need not to be true anymore, which may be surprising for the intuition 
and which has been a source of errors in some biological works. 

Moreover, if x* is an ES but Assumption (|4.16|) (a + c 7^ 0) is not true, Point (ii) of 
Proposition 14.131 may also fail. Indeed, in the case where a(x,x)d\2a(x,x) 7^ d\a{x, x)d2d(x, x) 
for x 7^ x* , 

_ , . . . r(x)d\20i(x,x) - r'(x)d2a(x,x) 

ni[x, x) + n2{x, x) - 



a(x, x)d\20t(x, x) — d\a(x, x)d2a(x, x) 
_ r(x*)(d 112 a(x*,x*) +d 12 2a(x*,x*)) - r'(x*)d 2 2a(x* ,x*) - r"(x*)d 2 a(x* ,x*) +o(l) 

a(x*, x*) (d\i2a(x* , x*) + di22®-(x* , x*)) — &2a(x* , x*)dua(x* , x*) — d\a(x*, x*)d22C*{x* , x*) + o(l) 

as x — > x* . This expression involves r"(x*), whose value is not imposed by the assumptions. 
Therefore, changing the function r in such a way that r(x*) and r'(x*) are fixed but r"(x*) 
changes also changes the value of\im. x ^ y ^ x * n\{x,y) +n,2(x,y). 



4.3.4 On triple coexistence in the neighborhood of x* 

Points (iii) and (iv) of Proposition 14.131 allow one to determine the signs of the 2- and 3- 
dimensional fitnesses in a trimorphic population with traits x,y,z close to x*. Combining 
this with the classification of Zeeman [30J (see Section l3T2l and Figure l3T2j) gives the following 
corollary. 

Corollary 4.15 For all ES x* satisfying the assumptions of Theorems 14.101 and such that 
duf(x*;x*) 7^ ; there exists a neighborhood hi ofofx* such that, for all distinct x , y , z € U, 
(x,y,z) C coex , where C coex is defined in (pTEL 
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Proof Let us assume for simplicity that x* = 0. We shall distinguish between the 
cases a > and a < 0, and prove in each case that the fitnesses cannot have any of the 
sign configuration corresponding to the classes 26, 29, 31 and 33 in the neighborhood of 
x* . Since all these classes contain the pattern of Fig. 13.11 we can assume without loss of 
generality that f(x; y) > 0, f(y; x) > 0, f(z; x,y) > and x < y. 

Consider first the case a < 0. It follows from Proposition l4.13l (iv) that the function /(•; •, •) 
has the shape of Fig. 14.11 (a) in the neighborhood of x*. In particular, this implies that 
x < z < y, f(z; x, y) > 0, f(x; y, z) < and f(y; x, z) < as soon as x, y, z are sufficiently 
close to x*. In view of Fig. 13.21 these conditions are incompatible with classes 31 and 33. 
Moreover, dnf(x;y) < for all x,y sufficiently close to x*. Therefore, by Lemma f4. 121 

d /ffayU = _ f 1 udnf{y + u{x _ y y)du (424) 
dx \ y-x J Jo 

is positive for all x, y sufficiently close to x*. Hence, since x < z < y, we have f(z; y)/(y — 
z) > f(x;y)/(y — x) > and thus f(z;y) > 0. Similarly, f(z;x) > 0. Together with 
f(z;x,y) > 0, these conditions are incompatible with classes 26 and 29. This ends the 
proof in the case where a < 0. 



f(z;x,y) 




(a) a < 0. (b) a > 0. 



Figure 4.1: The shape of the 3-dimensional fitness as a function of the sign of a. 

In the case where a > 0, by Proposition 14.131 (iv), /(•; ■) has the shape of Fig. 14.11 (b) 
in the neighborhood of x*. Therefore, z £ [x,y]. Assume for example that z < x < y. 
By Proposition 14.131 (iv) again, f(x;y,z) < and f(y;x,z) > 0. These conditions are 
incompatible with class 33. Moreover, using the fact that d\\f{x;y) > for all x,y 
sufficiently close to x* , it follows from the fact that (I4.24p is negative that f (z; y) / (y — z) > 
f(x;y)/(y — x) > and thus that f(z;y) > 0. Similarly, because of Assumption (I4.15p . 
d22f(x;y) > for all x,y sufficiently close to x*. Therefore, by Lemma f4. 121 

d //(y^h = _ r 1 ud22f{y . y + u{x _ y))du < o 

ox \ y — x / Jo 

for all x,y sufficiently close to x* . Thus, f(y;x) > implies that f(y;z) > 0. Together 
with the fact that f(x;y,z) < 0, these conditions are incompatible with classes 26, 29 and 
31. 

In the case where x < y < z, the method above proves that f(x;z) > 0, f(z;x) > and 
f(y;x,z) < 0, which is again incompatible with classes 26, 29, 31 and 33. This ends the 
proof of Corollary 14.151 □ 
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4.3.5 Double coexistence region in the neighborhood of x* 



We prove here Proposition EjTj that gives a criterion for the coexistence of two traits in the 
neighborhood of x* , and we end the proof of Theorem 14,101 The proof of Proposition 14.111 
is based on the study of the region of double coexistence, defined as {(x, y) G X : f(x; y) > 
and f(y;x) > 0} in the neighborhood of x* . The proof of Theorem 14.101 is based on a 
case-by-case study that extends the proof of Corollary 14.151 

Proof of Proposition 14.1 J It follows from Proposition 14.131 (iii) that the set of (x,y) £ 
X such that f(y; x) = is composed of the line {y = x} and of a set which is, because of 
the Implicit Function Theorem, a curve in the neighborhood of x*, containing (x*,x*) and 
admitting as tangent at this point the line {a(y — x*) = c(x — x*)}. Let us call 7 this curve. 
Since a < c, the curves 7 and {y = x} divide X 2 in the neighborhood of (x*,x*) into 4 
regions. Moreover, because of (|4.21l) . f(y;x) changes sign when the point (x, y) changes 
region by crossing either the line {y = x} or the curve 7. 

It is elementary from a case-by-case study to check that coexistence can occur in the 
neighborhood of x* if c > a > 0, a > c > 0, —a < c < < a and a < < —a < c, and 
that coexistence cannot occur in the neighborhood of x* if c < —a < < a, c < a < 0, 
a < c < and a < < c < —a. The cases where coexistence is possible are represented 
in Fig. 14.21 in the case where x* = 0. In these figures, the curve 7 is represented by its 
tangent line {a(y — x*) = c(x — x*)} and the sign of f(y;x) is represented by + and — signs 
depending on the position of (x,y) with respect to 7 and {y = x}. The sign of f(x;y) 
is obtained by an axial symmetry of the figure with axis {y = x}. We denote by 7 s the 
symmetric of the curve 7 with respect to this axis. The region of coexistence is the one 
where f(y; x) > and f(x; y) > 0. 

Note that the expansion of f(y;x) of Proposition 14.131 (iii) does not make use of any 
assumption on a and c. Therefore, a similar study can be done to treat the degenerate 
cases. One easily obtains that coexistence is possible in the neighborhood of (x*,x*) if 
c = a>0, c = and a > or a = and c > 0. Similarly, coexistence cannot occur in 
the neighborhood of (x* , x*) ifc = a<0, c = and a < or a = and c < 0. The case 
c = —a is undetermined and depends on higher-order expansions of the fitness function. □ 

Proof of Theorem 14.101 (b): case a < 

It follows from Theorems 14.61 that for any fixed 77 > 0, for e small enough, the PES 
stays monomorphic until it reaches (x* — rj, x* +77). Moreover, in view of the proof of 
Proposition 14.81 no mutant out of (x* — 77, x* + 77) can invade the population as long as it 
is monomorphic with support inside this interval. 

Now, by Proposition 14.111 when a < 0, coexistence may happen in the rescaled PES if 
c > —a. In this case, at the first coexistence time r e , the two traits x and y that coexist 
belong to (x* — 77, x* + 77) and are distant of less than eDiam(X) since m{x, •) has support 
in X — x. 

Let us examine what happens when a mutant trait z invades this population. Remind that 
we showed in the proof of Corollary 14. 151 that, if a < 0, x < y, f(x; y) > 0, f(y; x) > and 
f(z;x,y) > 0, then f(x;y,z) < 0, f(y;x,z) < 0, f(z;y) > and f(z;x) > 0. Examining 
Fig. 13.21 we see that these conditions are incompatible with all classes except classes 7 and 
9. Therefore, once the mutant z invades, the new state of the rescaled PES can be either 
n(z)8 z in the case of class 7, or either ni(x,z)5 x + n2(x,z)5 z or n\{y,z)8 y + fi2{y,z)5 z in 
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the case of class 9. In particular, we see that either the population becomes monomorphic 
again, or it stays dimorphic, but the distance between the two traits of the support of the 
PES has decreased. In addition, in both cases, the support of the new state of the PES is 
a subset of (x* — 77, x* + 77). Hence, 77-branching, as defined in Definition 14.91 cannot occur 
as soon as e < r] / (2Diam(X)) . This ends the proof of Theorem 14.101 (b). □ 

Proof of Theorem 14.101 (a): case a > 

By Proposition 14.111 when a > 0, under the assumptions of Theorem 14.101 we are in the 
situation of Fig. 14.21 (a), and hence coexistence is always possible in the neighborhood of 
x* . Fix 77 > 0. We are going to prove that, if 77 is small enough, then for e small enough, 

(i) the first time of coexistence r £ is a.s. finite and Supp(Z £ e _) C (x* — 77, x* + 77) a.s.; 

(ii) after time r £ , the distance between the two points of the support of the rescaled PES 

is non-descreasing and becomes a.s. bigger than 77/2 in finite time, before exiting the 
interval (x* — 77, x* +77). 



These two points will clearly imply Theorem 14.101 (a). 

For Point (i), observe first that, by Proposition 14.81 if r e < +00, then Supp(Z £ e _) C 
(x* — 77, x* + 77). Thus we only have to prove that F(r e < 00) = 1. 

In view of Fig. 14.21 (a), we observe that for a given jump size, the closer the support is 
from x* , the easier co-existence is. The proof is based on this fact, taking into account the 
additional difficulty that the jump rate is almost zero in that case. 
Fix k > 0. Let us define 

9 K = inf {t > : Supp(Z £ ) C {x* - ne,x* + ke)}. 

From Assumptions (A'3) and (A'"), the functions 

r+00 f-0 
ih I hm(x,h)dh and x 1— > / hm{x,h)dh 
Jo J -oo 

are continuous and there exists (3 > such that, for all x € [x* — 77, x* + 77], 

p+00 f-0 

/ h m(x, h)dh > (3 > and / hm(x,h)dh < -(3 < 0. (4.25) 

J0 J-00 

It is thus elementary to check, using (|4.21l) . that for any x € [x* — rj,x* — ne], resp. 

X € [x* + KE, X* + T?] , 

+00 

[g(x + eh, x)] + 7ti(x, h)dh > Ce 2 [3k > ; 

(3/2 

resp. / [g(x + eh,x)] + m{x,h)dh < —C'e /3k < 0. 
J — 00 

Assume that P(r e = 00 ; 0p/2 = 00) > 0. Then, on this event, in view of (14.1(1 . the 
previous inequalities show that there are infinitely many jumps in the TSS, with jump 
size bigger than e(3/2. This yields a contradiction since the TSS is monotonous before r e . 
Indeed, drawing a vertical line at some level x in Fig. 14.21 (a) (for example the vertical 
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dotted line), one can see that all the mutants invading the monomorphic population with 
trait x either coexist with x or are closer to x* than x. On the other hand, it is clear 
from Fig. 14.21 that the first jump after time #g/ 2 in the TSS with jump size bigger than 
e(3/2 (which almost surely happens) drives the TSS in the coexistence region. Therefore, 
P(r £ = oo ; 9/3/2 < oo) = and then P(r £ = oo) = 0. 

For Point (ii), assume that the rescaled PES is dimorphic at some time i, with support 
{x, y}, x < y. Let us examine what happens when a mutant trait z invades this population. 
Remind that we showed in the proof of Corollary 14.151 that, if a > and x,y,z belong 
to (x* — r]Q,x* + ?]o) for some % > and satisfy x < y, f(x;y) > 0, f(y;x) > and 
f(z;x,y) > 0, then 

• either z < x < y and f{x;y,z) < 0, f(y;x,z) > 0, f{z;y) > and f{y;z) > 0, 

• or x < y < z and f(x; y, z) > 0, f(y; x, z) < 0, f(z; x) > and f(x; z) > 0. 

We can assume without loss of generality that r/ < t]q. Examining Fig. 13.21 we see that 
both situations are only compatible with classes 9, 10, 11 and 12. Therefore, once the 
mutant z invades, the new state of the rescaled PES is n\(x, z)5 x + n^ix, z)5 z if x < y < z 
or fii(y, z)5 y + fi2(y, z)5 z if z < x < y. In both cases, we see that the distance between the 
two traits of the support of the PES can only increase until the stopping time 6' where one 
of the points of the support leaves (x* —r),x*+r]). In order to end the proof, it suffices to 
prove that, if rj is sufficiently small, 

9' < oo a.s. and Diam(Supp(Z|/)) > rj/2. 

The fact that 9' < oo a.s. can be proved using (|4.25p in a similar way as for Point (i). The 
lower bound of the diameter of the PES immediately follows from the fact that 

e Q u 2cDiam(X) 

t > t) K) a.s., where kq = . 

c — a 

This inequality follows from the following argument: for any x,y € M such that 

\ x - x *\ > 2c \ x ~y\ j (4.26) 
c — a 

it can be easily checked that 



:>l(l + -)\x-x*\ and \y - x*\ < (l + ° 



2 V c/ 1 1 1 ~ V 2c 

Since < a < c, we have 

1/ a\ a c — a c — a c 

-(! + ->- and 1 + — — < 1 + 



c 2c a a 

Now, {(y — x*) = (c/a)(x — x*)} is tangent to 7 at (x*,x*) and {(y — x*) = (a/c)(x — x*)} 
is tangent to 7 s at (x*,x*). Therefore, in view of Fig, l4~2l (a). any x,y G M satisfying (|4.26h 
do not coexist together. 

To conclude, it suffices to observe that, in the rescaled PES Z, the distance between a 
mutant trait and the trait of its progenitor in the PES is always smaller than eDiam(A'). 
Therefore, for any x £ (x* — r],x* + rj) such that \x — x*\ > ekq, any mutant trait y born 
from x do not coexist with x. □ 
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A Proof of Theorem [277 



The proof of this result is very similar to the proof of [U Thm.l]. We will not repeat all the 
details and we will restrict ourselves to the steps that must be modified. The general idea 
of the proof follows closely the heuristic argument of Section 12.4,11 Its skeleton is similar 
to the one in [4] for monomorphic populations. 
For all e > 0, t > 0, and T C X measurable, let 

A £t d(t,T) := |Supp(z^/#- Uif ) C T has d elements that coexist, say x\, . . . ,Xd, 

and VI < i < d, \(v t /Ku h : 1 {x i }) ~ "4 X )I < e}- 
To prove Theorem 12.71 we establish that for all e > 0, t > and r C X measurable, 

lim P(^ ed (i,r)) = P(Supp(Z t ) C r and has d elements). (A.l) 

where (Z t , t > 0) is defined in Theorem l2.71 The first ingredient of the proof is the following 
proposition, which generalizes Theorem 3 (a) and (b) of [4]. 

Proposition A.l Assume that, for any K > 1, Supp{vq) = {x\, . . . ,Xd} and (vq , li Xi \) € 
C a.s., where C is a compact subset o/M+. Let 4>(t, (m, . . . , n^j) denote the value at time 
t of the solution of LV(d, x) with initial condition (n\, . . . , rid). Then, for all T > 0, 

lim sup 

K ^+ OD l<i<d, t£[0,T] 

This result is a direct corollary of Theorem 11.2.1 of [11], except for two small difficulties. 
The first one is that Theorem 11.2.1 of |llj assumes that the function n ^ F x (n) involved 
in the definition 1|2.4|) of the Lotka Volterra system is uniformly Lipschitz on M+, which is 
not the case. However, observe first that, if rii < M for some M > for all i 6 {1, ... , d}, 
then (j)i(t, (ni, . . . , n^)) < M V (2A/a) for all t > 0. Indeed, if there is equality for some 
t > and i S {1, . . . , d}, then <fii(t, (n\, . . . , n^)) < 0. Therefore, the coefficients of the 
system LV(d, x) are uniformly Lipschitz on the set of states that can be attained by the 
solution of the system starting from any initial conditions in a compact set. The second 
difficulty is that Theorem 11.2.1 of [11] only implies that (1A.2|) holds on the event where 
there is no mutation between and T. In Lemma 2 (a) of [4], it is proved that for general 
initial condition Vq , the probability of mutation on the time interval [0, T] converges to 0, 
thus the conclusion follows. 

The second ingredient is the following exponential deviation estimate on the so-called 
"problem of exit from an attracting domain" |13j . It generalizes Theorem 3 (c) of [4]. 

Proposition A. 2 Let x\,...,Xd € X coexist. Then there exist constants c, V > such 
that, for any sufficently small e > 0, if ((vq , l{xA))i<i<d belongs to the (e/ '2) -neighborhood 
o/n(x), the time of exit of ({v^ , l{xi}))i<i<d from the e-neighborhhod of n(x) is bigger 
than e VK A r with probability converging to 1, where r denotes the first mutation time. 
Moreover, the previous result also holds if, for all i S {l,...,d}, the death rate of an 
individual with trait x\ 

d 

K x i) + ^2at(x i ,x j )(vf , 1 {X]} ) (A. 3) 

3=1 



(uf,l {xi} )-^(t,((^,l {xi} ),...,(^,l {Xd} ))) =0 a.s. (A.2) 



K 



K 
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is perturbed by an additional random process that is uniformly bounded by ce. 

Such results are fairly standard and can be proved in a variety of ways. We let the proof to 
the reader. The first part of this proposition is used to prove that, when the first mutation 
occurs, the population densities have never left the e-neighborhood of n(x) and the second 
is used to prove that, after the first mutation, as long as the mutant population is small, 
the resident population densities do not leave the e-neighborhood of n(x). In this case, the 
additional term in t) A.3[) is a(xi,y)(u^ , l{ y }), where y is the mutant trait, which is smaller 
that ae if (v^ ', !{,,}} < £• 

From these two results can be deduced the following lemma, which is the extension of 
Lemma 2 (b) and (c) of [4j. The proof is a simple copy of the argument in [4]. 

Lemma A. 3 Let Supp(vq) = {x±, . . . , Xd} that coexist and let r denote the first mutation 
time. There exists Eq such that, ifdvff, l{ Xi }))l<i<d belongs to the E^-neighborhood o/n(x) ; 
then, for any e < Eq, 

lim f(t > log if, sup \{vf , l {x . } ) - ni(x)\ < e) = 1, 

K^+co \ l<i<d, t6[logJf,r] 7 

d 

Kukt =^ Exp( y p(x,)A(x,)n,(x) ) 
i=i 

and lim F(at time r, the. mutant is born from trait Xi) — 1 ' ' ' 



for all i G {1, ... ,d}, where ==> denotes the convergence in law of real r.v. and Exp(u) 
denotes the exponential law with parameter u. 

The fourth ingredient is the following lemma, which is an extension of Lemma 3 of [4| . 

Lemma A. 4 Let Supp(u^) = {x\, . . . , Xd, y} where x±, . . . , Xd coexist and y is a mutant 
trait that satisfy Assumption (B). Let r denote the first next mutation time, and define. 

n = -mf{t > : Vi G /(n*), , l {xi} ) - n*\ < e and Va G* J(n*), (v? , l {x . } ) = 0} 
r 2 = mi{t > : (v? , l {y} ) = and V* G {1, . . . ,d}, \{vf , l {Xi} ) - n;(x)| < s}. 

Assume, that = 1/K (a single, initial mutant). Then, there exists Eq such that 

for all e < e$, if {{vq , ^-{ Xi }))i<i<d belongs to the e-neighborhood o/n(x) ; 

lim P( T1 < r 2 ) = %^1±, hm Pfa < n) ~ 1 



and Mi] > 0, lim P Ti A r 2 < — - — A r 

K-*+oo \ Kuk 

The proof of this lemma is similar to the proof of Lemma 3 in [4]. The main steps are the 
following. Assume first that e < 1/2. We introduce the following stopping times: 

R? = inf{t > : 3i G {1, ... ,d}, \{v* , l {xi} ) - n;(x)| > e} 
(v*,l {y} )>s} 
{u t K ,l {y} )=0}. 



Sf = inf {t > 
= inf {t > 
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i?g is the time of drift of the resident population away from its equilibrium, Sf is the 
time of invasion of the mutant trait (time t\ in Fig. 12 . 3[) and Sq is the time of extinction 
of the mutant trait. By the second part of Proposition IA.21 it can be proven exactly as 
in [4] that there exists p, V > and c < 1 such that, for K large enough, 

< t) > 1 - £ and P(Sf A r A e KV < R* /c ) > 1 - e. 



Kuk 

Then, on [0, r A Sf A R^ c ], by computing lower and upper bounds on the death rate of a 
mutant individual, it can be easily checked that, for K large enough, almost surely, 

ft / /..A" -i \ <r ft 

where, for i = 1 or — 1, Z l,e is a continuous-time branching process such that Zq £ = 1 and 
with birth rate (1 — ie)X(y) and death rate 

d 

Viu) + a (y> Xi)fii(x.) + i(d + l)a-. 
f-f c 

Next, we use the results of Theorem 4 of [4] on branching processes in order to control 
the probability that Z' l,£ /K exceeds e before it reaches 0, and to upper bound the time at 
which one of these events happens. As in [4], we obtain that there exists C > such that, 
for all rj > 0, e > sufficiently small and K large enough, 

*<' A j&MfA^ 2 l-I«g!±- 0l (A.4) 

On the event {5f < r A Sq A i^J, we introduce for e' > the stopping times 

if = mf{t > Sf : V* € {1, ... , d}, |<i/f , - n*| < and |<if , l {y} ) - n* d+1 \ < e 2 }, 

U« e , = inf{t > if : 3i £ I(n*), |(*f , - <l > e'} 

tf = inf{t > if : 3i I(n*), (if, > s}. 

We next use the Markov property at time Sf and apply Proposition IA. II as in [4] to obtain 
that there exists C > C such that, for K large enough, 

Pfsf < If < r A > [/( ^ x)]+ - C'e. 



Ku K ' X(y) 

Next, we can use again Proposition IA.2I to prove that there exists V > 0, C" > C and 
d < 1 such that 



P(Sf < Tf < V* A r A e xv " < tf £/c ,) > [/( ^ )]+ - C"e. 
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In a last step, we can as before prove that, for all t £ [T* , U^ e/c ,AV £ K ] and for all i I(n*), 



e,e/c' 



K ' 

where Z t,e is a continuous-time branching process such that Zljf K = \e 2 K~\ and with birth 
rate \{xj) and death rate 

£ 



f-i(xi) + ^ a(xi, Xj)n* - Card(/(n*))a— . 
j<=I(n*) 



Since, by Assumption (B2), /(xj;x*) < 0, this branching process is sub-critical if e is 
small enough. Hence, with arguments similar to the ones in |4] (especially the results of 
Theorem 4), we can prove that there exist C" > such that, for all 7] > 0, e > sufficiently 
small and K large enough, 

q k ~ „ ^ _ A v a tjk \ ^ [/(y; x )]+ w» 

D e < Ti < r A — A IS. > C e. 



KU K c,s/c>J - X{y) 

Combining this with (|A.4|) . we obtain Lemma lA.41 by letting e go to 0. 

Finally, (lA.lj) is deduced from these lemmas exactly as in [4] and similarly, the proof of 
Theorem O from (TO) . □ 
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(c) a > c> 0. (d) a < < -a < c. 

Figure 4.2: In the four cases where coexistence is possible, these figures show the sign configuration 
of f(y; x) depending on the position of (x, y) with respect to the curve 7 and the line {y = x} and 
the region of coexistence. For convenience, we assumed x* = 0. 
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